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ABSTRACT 



This paper discusses the efforts of the Department of 
Mathematics at the United States Coast Guard Academy (USCGA) to determine the 
degree to which their courses support the published academic outcomes of the 
institution, and presents the results of a survey of student attitudes toward 
the academic outcomes . A survey questionnaire was developed and administered 
to 425 students taking mathematics courses during the spring term of 1997. 
Factor analysis of the results led to the identification of five outcome 
factors: problem solving, writing, reading, applied learning, and evaluation. 
The construct validity of the instrument was also examined. It is concluded 
that the construction of the survey required the department to examine in 
detail how its courses supported published USCGA outcomes and provided 
valuable feedback in the curriculum development process by establishing 
baseline assessments for each outcome. Three appendixes provide a listing of 
judgmental categories for attitude toward academic outcomes in mathematics 
courses, a copy of the survey questionnaire, and statistical tables. (MDM) 
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Abstract 

Assessment of academic outcomes is an essential component of curriculum development. The Dean of 
Academics at the United States Coast Guard Academy has published 10 academic outcomes which describe a broad 
range of behaviors and attitudes representing an ideal product of the 4-year academic program. This paper discusses 
the efforts of the Department of Mathematics to determine the degree to which their courses support the published 
academic outcomes. A survey instrument measuring student attitudes towards the academic outcomes was 
developed and administered to 425 students taking mathematics courses during the Spring '97 semester. 

Exploratory common factor and Rasch latent trait analyses were used to examine construct validity. Alpha 
reliabilites were calculated. The factor analysis resulted in five outcome factors: Problem Solving, Writing, 

Reading, Applied Learning and Evaluation. The results of the study will be used to illustrate how the Department 
of Mathematics will establish a baseline assessment of student attitudes towards academic outcomes for each course. 
Establishing baseline assessments of student attitudes will allow llie Department to identify changes in student 
attitudes which can be used in evaluating the effectiveness of modifications to instructional format and/or curriculum. 
The study illustrates how student attitudes can be used in the assessment process to provide valuable information for 
curriculum development. 



Introduction 

The United States Coast Guard Academy is one of five federal service academies. The 
student body consists of approximately 850 students from all over the United States, as well as a 
dozen or so exchange students from countries around the world. The Coast Guard Academy 
provides the opportunity for students to obtain a Bachelor of Science degree in one of eight 
majors, in addition to a commission as an Ensign in the United States Coast Guard. As an 
educational institution, the Coast Guard Academy considers the issues of academic outcomes and 
the assessment of those outcomes to be very important. Over the last several years the 
Academic Division has engaged in a great deal of dialogue about the academic program and what 
graduates should be getting from that program. As a result of this dialogue the Dean has 
published 10 academic outcomes which define the purpose of the academic program. These 
outcomes are listed below. 

A cadet shall be able to: 

1. read and understand a variety of written materials, listen critically to oral 
arguments, and formulate penetrating questions. 



1 Paper presented at the annual meeting of the Northeastern Educational Research Association, October, 1997, 
Ellenville, NY. 
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2. write clear, concise, persuasive and grammatically correct passages on general or 
professional topics, from a paragraph to several pages in length. 

3. apply the basic skills of critical analysis, quantitative reasoning and problem 
solving to complex tasks in a broad range of contexts. 

4. prepare and deliver a well organized and polished oral presentation to a variety of 
audiences on topics within their fields of competence. 

5 gain access to a broad range of information systems and locate desired data 
reliably. 

6. integrate knowledge and information efficiently into a working conceptual 
framework that lends itself to continued expansion and refinement. 

7. show evidence that they (graduates) are capable of honest, realistic, and 
constructive self-evaluation, that they can devise successful and creative strategies 
to develop their strengths and correct their weaknesses, and that they demonstrate 
the intellectual, moral, and physical stamina to follow through. 

8. function effectively as a member of a team or working group that is charged with 
studying a complex problem or a significant policy issue and arriving at a solution 
or recommendation. 

9. comprehend the interrelationship of the diverse social, economic, political, 
cultural, technological, and environmental forces that shape the world in which the 
Coast Guard operates. 

10. articulate their personal values and those of the Coast Guard and public service in 
general, recognize conflicts in value systems when they exist, and formulate 
reasoned arguments to support their resolutions of the conflicts. 

These outcomes, by describing a broad range of behaviors and attitudes, represent an ideal 
product of the 4 year academic program at the Coast Guard Academy. Assuming that these 
outcomes adequately describe the purpose of the academic program, the question becomes how 
are these outcomes assessed? Does the academic program at the United States Coast Guard 
Academy achieve its stated goals? At the start of the 1996-97 academic year the Dean tasked 
each academic department to investigate possible answers to this question. 

The Mathematics Department approached this task by first identifying specific activities 
within each of its courses that seemed to support the academic outcomes. For example, in 
Calculus-I cadets are assigned daily reading assignments from the text for which they are held 
accountable on tests, quizzes, worksheets, and projects. Faculty members felt that this activity 
most directly supported outcome 1. A list of activities was developed for each of the courses 




4 



Page 2 



taught by the Department. In reviewing the lists of activities it became obvious that student 
grades on homework, projects, presentations, and exams provided one measure of success. These 
grades were evaluating the success of the course and students from a faculty point of view. An 
equally important point of view is that of the students. This lead to the development of an 
instrument which measures student attitudes towards the academic outcomes. This instrument 
measures how the students felt the courses they were taking supported the stated academic 
outcomes. Such an approach is consistent with that presented by Pike (1992): 

It is time for institutions to consider alternative approaches to assessing general education 
outcomes. The approaches will, of necessity, be institution-specific, be frequently 
qualitative in nature, and seek information about the effectiveness of general education 
programs based on faculty and student perceptions of those programs, (p. 157) 

Methodology 

Survey Development 

Constructing Items. The course activity lists for all departmental courses were reviewed and 
translated into specific student objectives through a series of discussions among all faculty in the 
department. This process produced a list of student objectives for each of the academic outcomes 
by course. Since the goal was to develop an instrument that could be used for all mathematics 
courses, a common core of objectives present across all mathematics courses was compiled. This 
common core of objectives was used to develop the survey instrument. The instrument 
consisted of the list of objectives, along with the question, "To what extent do you think this 
course helps you to accomplish the following objectives?". Responses were obtained on a 5- 
point Likert scale ranging from Very Little to A Great Deal. In reviewing the list of objectives it 
became quite obvious that not all academic outcomes were equally supported by the mathematics 
courses. Only those outcomes for which five or more objectives had been identified were selected 
for inclusion in the survey. Outcome numbers, 1, 2, 3, 6 and 7, would be the constructs which 
the survey would be designed to measure. 

Content Validity. The first step in reviewing the instrument design was to examine content 
validity. Content validity is meant to answer the question, To what extent do the objectives on 
the instrument adequately sample from the intended universe of content? In other words, do the 
identified objectives really support the stated academic outcome? The objectives (items) were 
randomly arranged and the 12 faculty in the Department of Mathematics were asked to identify 
by number the outcome which the objective supported. Those items having percent agreement of 
82% or better were kept on the instrument. After completing the judgmental review process, 



several outcomes had less than six objectives associated with them. In an effort to have at least 
six items per outcome, a focus group consisting of five mathematics faculty members convened to 
discuss the addition of new items, as well as how unsatisfactory items could be rewritten to 
better represent a particular outcome. The results of the judgmental review and focus group are 
summarized in Appendix A. 

Sample Characteristics. After several iterations of the survey were reviewed by the 
mathematics faculty a final form was adopted (See Appendix B for a copy of the final 
instrument.) The survey was administered to 425 cadets who were taking mathematics courses 
during the Spring 1997 semester. The characteristics of the sampled cadets are summarized in 
Tables 1-5. It is important to note that a cadet taking more than one mathematics course during 
the spring semester would complete a survey in each course. 

Table 1 



Frequency Breakdown of Students by Course (n=425) 



Course 


Count 


Percent 


Calculus I 


58 


14 


Calculus II 


164 


38 


Calculus III 


37 


9 


Probability & Statistics 


52 


12 


Differential Equations 


68 


16 


Probabilistic Models 


33 


8 


Numerical Analysis 


12 


3 


Missing Information 


1 


<1 



Table 2 



Frequency Breakdown of Students by Major (n-425) 



Major 


Count 


Percent 


Civil Engineering 


72 


17 


Electrical Engineering 


30 


7 


Mechanical Engineering 


71 


17 


Navel Architecture 


40 


9 


Operations Research 


65 


15 


Marine Science 


57 


13 


Government 


54 


13 


Management 


35 


8 


Missing Information 


1 


<1 
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Table 3 



Frequency Breakdown ofStudents by Gender (n=425) 



Gender 


Count 


Percent 


Female 


118 


28 


Male 


304 


72 



Table 4 

Frequency Breakdown of Students by Class Year. (n=425) 



Major 


Count 


Percent 


Senior 


6 


1 


Junior 


43 


10 


Sophomore 


130 


31 


Freshman 


243 


57 


Missing Information 


3 


<1 



Table 5 

Histogram of Cumulative Grade Point Average (n=425) 




cumulative grade point average 



Construct Validity. The construct validity of the instrument was also examined. Construct 
validity answers the question, To what extent do certain explanatory concepts explain covariation 
in the responses to the items on the instrument ? In other words, did the students answering the 
survey respond in a similar fashion to all the items that belong to a particular category (outcome). 
Construct validity was investigated by using Common Factor Analysis. Common Factor 
Analyses with both varimax and oblique rotations were conducted. The oblique rotation provided 
very similar results as the varimax rotation. Factor names were based on the oblique rotation. 
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since it provided a clearer result in addition to providing information about the intercorrelations 

among the factors. 

Table 6 

Common Factor Analysis with Oblique Rotation: Attitude Toward Academic Outcomes in 
Mathematics Courses. ( h=425 ) 



Factor 


Item 

Number 


Stem 


Loading 


Factor I 


20 


Select one of several possible solution techniques for a given 


.73 


Problem Solving Outcome 


21 


mathematical problem. 

Break a large complex problem into a series of smaller sub- 


.71 




11 


problems. 

Infer a solution to a given problem by making use of 


.54 




2 


solutions from smaller sub-problems. 

Given several possible solution techniques for a 


.49 




24 


mathematical problem select the best one. 

Use ail alternative solution technique to verify a problem 


.42 




12 


solution. 

Use course material to solve problems in many different 


.41 




32 


contexts. 

Synthesize "new” results by combining previously 


.31 



established results along with general mathematical 
principles. 



Factor II 

Writing Outcome 


9 

28 

22 


Write clear explanations of mathematical concepts as 
responses to short answer questions on exams. 

Produce short written responses, in paragraph form, 
explaining various mathematical concepts. 

Present written justifications of problem solutions in an 
understandable manner. 


.58 

.50 

.47 


Factor III 

Reading Outcome 


27 


Formulate questions concerning mathematical concepts from 
reading the textbook. 


.68 




31 


Read mathematical concepts in your textbook. 


.66 




13 


Explain a mathematical concept to a peer or instructor based 
on your reading of the textbook. 


.63 




23 


Answer questions concerning course material 
extemporaneously in class, based on your reading of the 
textbook. 


.61 


Factor IV 

Applied Learning Outcome 


3 


Identify real world applications of the course content . 


0.83 




34 


Identify 7 how skills learned in this course will be useful to 
you in the future. 


0.73 




16 


Connect course material to information learned in other 
courses. 


0.48 


Factor V 

Evaluation Outcome 


4 


Recognize when you need to seek extra help with course 
material. 


0.62 




5 


Conduct self-assessments of your understanding of course 
material. 


0.59 




10 


Identify your strengths and weaknesses with the course 
material. 


0.59 




38 


Take notes in class which are clearly written and useful for 
further study. 


0.37 




26 


Develop successful strategies for learning the course material. 


0.31 
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Factor Names. The following names were given to the factors. 

Factor I was named the Problem Solving Outcome because the items which had the 
highest correlations with the factor dealt with problem solving. Cadets with a high score on this 
scale feel that the mathematics course they are taking greatly contributes to their ability to select 
problem solution techniques, as well break problems up and infer a solution to the original 
problem based on the solutions of the smaller problems. 

Factor II was labeled the Writing Outcome because the items which had the highest 
correlations with the factor dealt with being able to write responses explaining mathematical 
concepts, justifying solution techniques or doing evaluations. Cadets with a high score on this 
scale feel that the mathematics course they are taking greatly contributes to their ability to 
produce short written responses explaining various mathematical concepts. This includes 
providing written justifications for problem solutions, as well as brief evaluations about various 
aspects of the course. 

Factor III was called the Reading Outcome because the items which had the highest 
correlations with the factor dealt with being able to read mathematical concepts from a textbook, 
explain mathematical concepts to peers or instructors based on reading as well as formulating 
questions based on reading. Cadets with a high score on this scale feel that the mathematics 
course they are taking greatly contributes to their ability to read and understand mathematical 
concepts from the textbook. 

Factor IV was named the Applied Learning Outcome because the items which had the 
highest correlation with the factor dealt with being able to identify real world applications of the 
course content as well as how the learned material would be useful in the future. Cadets with a 
high score on this scale feel that the mathematics course they are taking greatly contributes to 
their ability to connect their learning to actual real world applications. 

Factor V was labeled the Evaluation Outcome because the items which had the highest 
correlation with the factor dealt with being able to recognize when they, the cadet, needed help 
with the course material by monitoring their performance as well as knowing their strengths and 
weaknesses. Cadets with a high score on this scale feel that the mathematics course they are 
taking greatly contributes to their ability to monitor and evaluate their understanding of course 
material. 

The derived factors, as noted in Appendix A, did agree with the judgmental categories 
which were established during the content validity phase of the study. It should be noted that 
there is a moderate correlation between the Problem Solving Outcome and the Applied Learning 
Outcome. This correlation was actually identified during the content validity phase of the 
instrument design. Many of the faculty members involved in the judgmental review commented 
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on the fact that they thought these two outcomes were very closely related. The correlations 
between the factors are contained in Table 7. 



Table 7 

Attitude Towards Academic Outcomes in Mathematics Courses Factor Intercorrelation Matrix 
Oblique Rotation (n=425) 





Problem 

Solving 

Outcome 


Writing 

Outcome 


Reading 

Outcome 


Applied 

Learning 

Outcome 


Evaluation 

Outcome 


Problem Solving 


1.0 










Outcome 












Writing Outcome 


.27 


1.00 








Reading Outcome 


.23 


.33 


1.00 






Applied Learning 


.49 


.34 


.35 


1.00 




Outcome 












Evaluation Outcome 


.39 


.20 


.18 


.31 


1.00 



Item Response Analysis Results. In addition to the exploratory factor analysis the Item 
Response Theory Scale program was run on each of the derived factors. This program is 
designed to provide Rasch Latent Trait Analysis information about the scales. Maps of the 
items and for people for each scale are presented in Appendix C. Gable, Ludlow, & Wolf (1990) 
show how the variable maps from the Scale program can be used in identifying weaknesses in the 
construct definition. For this paper the reader should focus on the vertical line ( i.e., difficulty 
continuum ) presented in the center of the map. The upper end of the continuum would contain 
items most difficult to rate “A Great Deal”, while the lower end would contain items that are 
easier to respond “A Great Deal”. The construct validity issue is reflected in how well the items 
are spread across the continuum. A restriction in the item locations regarding the spanning of the 
continuum would suggest that the scale could not adequately differentiate the opinions or 
attributes of a high scoring person from those of a low scoring person. The maps presented in 
Appendix C indicate that the items on the Problem Solving Scale and the Evaluation Scale are 
adequately spread across the continuum that is being measured. However, the map for the 
Reading scale shows that there are two clusters of items, items 13 and 23 have an item value of 
0.19 and items 27 and 31 have an item value of -0.16. This indicates that these four items are 
actually only measuring two locations on the continuum. The map for the Writing scale also has 
two clusters of items, items 9 and 22 have an item value of -0.24 and item 28 is spread away 
with an item value of 0.39. The Applied Learning scale has items 16 and 34 together with an 
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item value of 0.09 and item 3 has a value of -0.24, thus showing some spread. The items 
positioned close together will need to be examined in future investigations. 



Reliability. The reliability coefficient gives an indication of the internal consistency of the 
responses to the items defining the scale. Reliability for the scales on this survey were calculated 
using the alpha internal consistency reliability coefficient. The reliability coefficients ranged from 
.69 for the Evaluation Outcome to .83 for the Problem Solving Outcome. In reviewing the 
reliability output only the Applied Learning Outcome has an item which, if deleted, would 
improve the reliability. The reliability information is summarized in Table 8. 



Table 8 



Summary of Altitude Toward Academic Outcomes, in Mathematics Courses. (N=425) 



Factor 


Item # 


1 


Response Percentages 
2 3 4 


5 


Mean 


Std. 

Dev. 


Corrected 
Item Scale 
Corr. 


Alpha 

Reliability 

Item 

deleted 


Alpha 

Relia- 

bility 


Problem 


20 


3 


11 


42 


37 


7 


3.35 


.86 


.63 


.79 




Solving 


21 


2 


13 


35 


42 


8 


3.40 


.89 


.61 


.79 




Outcome 


11 


2 


7 


35 


45 


11 


3.55 


.86 


.57 


.80 






2 


1 


12 


33 


45 


9 


3.50 


.84 


.51 


.81 


.83 




24 


4 


20 


45 


27 


4 


3.07 


.89 


.56 


.81 






12 


4 


14 


44 


33 


5 


3.23 


.88 


.54 


.81 






32 


4 


13 


48 


31 


4 


3.18 


1.05 


.58 


.80 




Reading 


27 


10 


27 


42 


17 


4 


2.77 


.98 


.57 


.76 




Outcome 


31 


11 


29 


37 


20 


3 


2.76 


1.00 


.63 


.73 


.79 




13 


17 


28 


36 


15 


4 


2.59 


1.06 


.60 


.74 






23 


13 


34 


38 


14 


1 


2.57 


.93 


.61 


.74 




Writing 


9 


7 


28 


41 


21 


3 


2.85 


.93 


.58 


.68 




Outcome 


28 


14 


35 


35 


14 


2 


2.56 


.96 


.58 


.68 


.74 




22 


6 


25 


44 


22 


3 


2.89 


.92 


.60 


.63 




Applied 


3 


5 


21 


33 


31 


10 


3.18 


1.05 


.62 


.61 




Learning 


34 


11 


23 


32 


27 


7 


2.97 


1.10 


.63 


.60 


.75 


Outcome 


16 


8 


24 


37 


24 


7 


2.95 


1.03 


.48 


.77 




Evaluation 


4 


3 


10 


29 


39 


19 


3.62 


.98 


.43 


.64 




Outcome 


5 


1 


13 


43 


35 


8 


3.34 


.86 


.50 


.62 


.69 




10 


1 


11 


33 


45 


10 


3.52 


.84 


.49 


.62 






38 


6 


10 


25 


32 


27 


3.63 


1.15 


.37 


.68 






26 


5 


17 


40 


32 


6 


3.19 


.94 


.46 


.63 
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Conclusion and Future Plans 



Survey Evaluation. Overall the initial results of the designed instrument appear very promising. 
The derived constructs corresponded very well to the targeted judgmental categories. The 
Problem Solving scale appears to be very sound as it provided reliable data and items 
corresponded well with the judgmental category for Problem Solving. The other scales need to 
have additional items added so that the reliability can be improved. To increase reliability to .80 
the Reading Scale needs to have one item added, the Writing Scale needs two items, the Learning 
Scale needs one additional item, and the Evaluation Scale needs four items. These new items 
when added to the valid items identified in this study would provide an instrument which 
contains a total of 30 items. 

Assessment Implications. The survey presented in this paper provides two important 
contributions to the assessment process. First, the survey development required the Department 
of Mathematics to examine in detail how its courses support the published USCGA outcomes. 
General academic outcomes are designed to provide direction for curriculum development without 
prescribing specific actions. Such general direction affords academic departments flexibility in 
curriculum development. However, it is often difficult to know if and how specific instructional 
methodologies support such general outcomes. The development of survey items along with the 
results of the factor analysis have provided a definition of the general outcome in terms of 
specific course activities. For example, the Problem Solving Outcome states, 

A cadet shall be able to apply the basic skills of critical analysis, quantitative reasoning and 

problem solving to complex tasks in a broad range of contexts. 



The survey results show that problem solving in mathematics course can be defined in terms of 
the following student behaviors. Cadets shall be able to: 

- Select one of several possible solution techniques for a given mathematical problem; 

- Break a large complex problem into a series of smaller sub-problems; 

- Infer a solution to a given problem by making use of solutions from smaller sub-problems; 

- Given several possible solution techniques for a mathematical problem select the best one; 

- Use an alternative solution technique to verify a problem solution; 

- Use course material to solve problems in many different contexts, 

- Synthesize "new" results by combining previously established results along with general 
mathematical principles. 
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This operational definition of problem solving provides more specific information for use in the 
instructional design process. 

The second contribution relates to the following statement by Hlebowitsh (1995) when 
he suggests 

The progressive idea of assessment in the curriculum is one that is continuous and that, in 
certain instances, might be an initiating activity rather than a concluding one in the 
conduct of curriculum improvement or reform. 

In light of this definition of assessment, the survey presented in this paper will provide valuable 
feedback in the curriculum development process by establishing baseline assessments for each 
outcome. These baselines can then be used to evaluate the effects of curriculum changes relative 
to the published outcomes. 

Future Investigations. In addition to improving the survey instrument, further investigations 
into differences in student attitudes by course, academic major and grade level would be 
appropriate. Such investigations may provide insights into the generalization of the survey 
instrument for use by other academic departments. 
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Appendix A 

Judgmental Categories for Attitude toward Academic Outcomes in Mathematics Courses 



Outcome 1: 

A cadet shall be 
critically to oral 


able to read and understand a variety of written materials, listen 
arguments, and formulate penetrating questions. 




Judgemental Review 
Percent Agreement 


Survey 
Item # 


Item Stem 


Derived 

Factor 


100% 


6 


Understand mathematical concepts from your textbook. 




100% 


25 


Read and understand mathematical theorems. 




100% 


27 


Formulate questions concerning mathematical concepts from 
reading the textbook. 


Reading 

(.68) 


91% 


13 


Explain a mathematical concept to a peer or instructor based on 
your reading of the textbook. 


Reading 

(.63) 


91% 


30 


Listen critically to oral presentations concerning mathematical 
concepts. 




91% 


31 


Read mathematical concepts in your textbook. 


Reading 

(.66) 


82% 


14 


Question mathematical concepts which are presented in class. 




82% 


23 


Answer questions concerning course material extemporaneously 
in class, based on your reading of the textbook. 


Reading 

(.61) 


64% 


Not 

Used 


Read mathematical problems and represent them 
with appropriate mathematical notation. 





Outcome 2: 

A cadet shall be able to write clear, concise, persuasive and grammatically correct 
passages on general or professional topics, from a paragraph to several pages in length. 



Judgemental Review 
Percent Agreement 


Survey 
Item # 


Item Stem 


Derived 

Factor 


100% 


22 


Present written justifications of problem solutions in an 
understandable manner. 


Writing 

(0.47) 


100% 


28 


Produce short written responses, in paragraph form, explaining 
various mathematical concepts. 


Writing 

(.50) 


91% 


1 


Provide a written problem solution using proper adherence to 
rules of logic. 




91% 


9 


Write clear explanations of mathematical concepts as responses 
to short answer questions on exams. 


Writing 

(.58) 


82% 


29 


Provide a written problem solution with precise use of 
mathematical symbols. 




55% 


Not 

Used 


Take notes in class which are useful for 

understanding and applying the concepts. 




18% 


Not 

Used 


Study from your classnotes as you prepare for an 
exam. 




9% 


Not 

Used 


Complete homework problems using your class 
notes. 




New Item added from 
focus group discussion 


36 


Write a brief evaluation about some aspect of the course, (i.e. a 
particular course topic or the instructor, etc.). 




Take notes item above 


38 


Take notes in class which are clearly written and useful for 


Eval 


rewritten during focus 
group discussion 




further study. 


(.37) 
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Appendix A (cont.) 

Judgmental Categories for Attitude toward Academic Outcomes in Mathematics Courses 



Outcome 3: 

A cadet shall be able to apply the basic skills of critical analysis, quantitative 
reasoning and problem solving to complex tasks in a broad range of contexts. 




Judgemental Review 


Survey 


Item Stem 


Derived 


Percent Agreement 


Item # 




Factor 


100% 


35 


Prove a mathematical theorem. 




100% 


37 


Solve mathematical problems without having to mimic a similar 
problem solution. 




91% 


18 


Apply appropriate mathematical techniques to solve problems 




91% 


21 


Break a large complex problem into a series of smaller sub- 
problems. 


Problem 

(.71) 


91% 


24 


Use an alternative solution technique to verify a problem 
solution. 


Problem 

(.42) 


82% 


11 


Infer a solution to a given problem by making use of solutions 
from smaller sub-problems. 


Problem 

(.54) 


73% 


Not 

Used 


Evaluate possible solution techniques for a given 
mathematical problem 




64% 


Not 

Used 


Use course material to solve problems in other 
courses. 




45% 


Not 

Used 


Solve problems in other disciplines such as physical 
science, management science, engineering, and 
social science using the course material. 




45% 


Not 

Used 


Discuss and justify the relative merits of different 
problem solving techniques. 




Rewrite of item by focus 


2 


Given several possible solution techniques for a mathematical 


Problem 


group 




problem select the best one. 


(.49) 


Rewrite of item by focus 


12 


Use course material to solve problems in many different 


Problem 


group 




contexts. 


(.41) 


New item by focus 


20 


Select one of several possible solution techniques for a given 


Problem 


£°HE 




mathematical problem. 


(-73) 



Outcome 6: 

Integrate knowledge and information efficiently into a working conceptual framework 
that lends itself to continued expansion and refinement. 



Judgemental Review 
Percent Agreement 


Survey 
Item # 


Item Stem 


Derived 

Factor 


91% 


3 


Identify real world applications of the course content . 


Learning 

(.83) 


91% 


7 


Use previously learned material to support new learning. 




82% 


16 


Connect course material to information learned in other courses. 


Learning 

(.48) 


82% 


32 


Synthesize "new” results by combining previously established 
results along with general mathematical principles. 


Problem 

(.31) 


82% 


33 


Understand the inter-relationship between the various 
mathematical concepts contained in this course. 




82% 


34 


Identify how skills learned in this course will be useful to you in 
the future. 


Learning 

(.73) 


73% 


17 


Explain the relationships between the major concepts covered in 
this course. 




64% 


15 


Identify the major topics contained in the course. 
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Appendix A (cont.) 

Judgmental Categories for Attitude toward Academic Outcomes in Mathematics Courses 



Outcome 7: 

Show evidence that they (graduates) are capable of honest, realistic, and constructive 
self-evaluation, that they can devise successful and creative strategies to develop their 
strengths and correct their weaknesses, and that they demonstrate the intellectual, moral, 
and physical stamina to follow through.. 


Judgemental Review 


Survey 




Derived 


Percent Agreement 


Item # 


Item Stem 


Factor 


100% 


5 


Conduct self-assessments of your understanding of course 
material. 


Eval 

(.59) 


100% 


8 


Know how you did on an exam prior to seeing the grade. 




100% 


10 


Identify your strengths and weaknesses with the course material. 


Eval 

(.59) 


91% 


26 


Develop successful strategies for learning the course material. 


Eval 

(.31) 


54% 


Not 

Used 


Investigate and justify the accuracy and correctness 
of your problem solving strategy. 




45% 


Not 

Used 


To know when a problem solution is incorrect. 




9% 


Not 

Used 


Develop an intuition about what the correct solution 
should be. 




9% 


Not 

Used 


Use an alternative solution technique to verify a 
problem solution. 




New item added by focus 


4 


Recognize when you need to seek extra help with course 


Eval 


group discussion 




material. 


(.62) 


New item added by focus 
group discussion 


19 


Recognize when you have adequately prepared for an exam. 
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Appendix B 



United States Coast Guard Academy 
Department of Mathematics 

Course Evaluation Form 

INSTRUCTIONS: 

Your responses to the following survey items will be recorded on a General Purpose -NCS- 
Answer Sheet. Please use only a No. 2 soft lead pencil. Complete the following identifying information 
on the left side of the NCS Answer sheet. Please insure that all identifying information has been 
properly completed with the appropriate bubbles darkened. 



- In the block titled NAME indicate your major or intended major by writing one of the below 
given two letter codes in the two left-most columns and darkening the appropriate bubbles. 

CE - Civil Engineering GT - Government 

EE - Electrical Engineering OR - Operations Research 

ME - Mechanical Engineering MS - Marine Science 

NA - Naval Architecture and Marine Engineering MT - Management 

- In the block titled SEX indicate your gender by darkening the appropriate bubble. 

- In the block titled GRADE or EDUC indicate your year group by darkening the bubble which is 
appropriate and corresponds to the following list. 

1 - FIRST CLASS CADET 

2 - SECOND CLASS CADET 

3 - THIRD CLASS CADET 

4 - FOURTH CLASS CADET 

- In the block titled BIRTHDATE put TODAY'S date. 

- In the block titled IDENTIFICATION starting with block A, put the 4 digit course number for 

the course you are enrolled in and darken the appropriate bubbles. 

- In the block titled SPECIAL CODES put your current CUMULATIVE GRADE POINT 

AVERAGE and darken the appropriate bubbles. For example a cumulative GPA of 3.45 
would be entered starting with block K as 345. 



Indicate your responses to the following question by darkening the appropriate bubble on the NCS 
answer sheet which corresponds with the item number. 
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Appendix B (cont.) 



United States Coast Guard Academy 
Department of Mathematics 

Course Evaluation Form 

QUESTION: To what extent do you think this course helps you to accomplish the following 
objectives. Respond using the following scale. 

A B C D E 

VERY LITTLE A GREAT DEAL 

1 Provide a written problem solution adhering to proper rules of logic. 

2 Given several possible solution techniques for a mathematical problem select the best 
one. 

3 Identify real world applications of the course content . 

4 Recognize when you need to seek extra help with course material. 

5 Conduct self-assessments of your understanding of course material. 

6 Understand mathematical concepts from your textbook. 

7 Use previously learned material to support new learning. 

A B C D E 

VERY LITTLE A GREAT DEAL 

8 Know how you did on an exam prior to seeing the grade. 

9 Write clear explanations of mathematical concepts as responses to short answer 
questions on exams. 

10 Identify your strengths and weaknesses with the course material. 

1 1 Produce a solution to a given problem by making use of solutions from smaller sub- 
problems. 

12 Use course material to solve problems in many different contexts. 

13 Explain a mathematical concept to a peer or instructor based on your reading of the 
textbook. 



A 

VERY LITTLE 



B 



C 



D 



E 

A GREAT DEAL 



Appendix B (cont.) 



United States Coast Guard Academy 
Department of Mathematics 

Course Evaluation Form 

QUESTION: To what extent do you think this course helps you to accomplish the following 
objectives. Respond using the following scale. 

A B C D E 

VERY LITTLE A GREAT DEAL 

14 Formulate questions about mathematical concepts presented in class. 

15 Identify the major topics contained in the course. 

16 Connect course material to information learned in other courses. 

17 Explain the relationships between the major concepts covered in this course. 

18 Apply appropriate mathematical techniques to solve problems. 

19 Recognize when you have adequately prepared for an exam. 

20 Select one of several possible solution techniques for a given mathematical problem. 



A B C D E 

VERY LITTLE A GREAT DEAL 

21 Break a large complex problem into a series of smaller sub-problems. 

22 Present written justifications of problem solutions in an understandable manner. 

23 Answer questions concerning course material extemporaneously in class, based on your 
reading of the textbook. 

24 Use an alternative solution technique to verify a problem solution. 

25 Read and understand mathematical theorems. 

26 Develop successful strategies for learning the course material. 

27 Formulate questions concerning mathematical concepts from reading the textbook. 
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A 

VERY LITTLE 



B 



D 



E 

A GREAT DEAL 



Appendix B (cont.) 



United States Coast Guard Academy 
Department of Mathematics 

Course Evaluation Form 

QUESTION: To what extent do you think this course helps you to accomplish the following 
objectives. Respond using the following scale. 

A B C D E 

VERY LITTLE A GREAT DEAL 

28 Produce short written responses, in paragraph form, explaining various mathematical 
concepts. 

29 Provide a written problem solution with precise use of mathematical symbols. 

30 Listen critically to oral presentations concerning mathematical concepts. 

31 Read mathematical concepts in your textbook. 

32 Synthesize new results by combining previously established results along with general 
mathematical principles. 

33 Understand the interrelationship between the various mathematical concepts contained 
in this course. 

^ 

VERY LITTLE A GREAT DEAL 

34 Identify how skills learned in this course will be useful to you in the future. 

35 Prove a mathematical theorem. 

36 Write a brief evaluation about some aspect of the course, (i.e. a particular course topic 
or the instructor, etc.). 

37 Solve mathematical problems without having to mimic a similar problem solution. 

38 Take notes in class which are clearly written and useful for further study. 



A 

VERY LITTLE 



B 



C 



D 



E 

A GREAT DEAL 



Appendix C 



423 PERSONS 



7 ITEMS 
ERROR 



SCALE: Problem Solving Outcome 
MAX. OF 5 CATEGORIES 28 STEPS 05/06/97 | 



PAGE 10 



SCORE PERSON 
{FREQ) POSITION 



MAP SHOWING POSITIONS OF PEOPLE AND ITEMS ON THE VARIABLE 
PEOPLE <N= 423) ITEMS (L= 7) 



ITEM 

VALUE (SE) FIT 



27 ( 


3) 


5.10 


1.07 


XXX] 

] 

] 

] 

1 






26 ( 


1) 


4.26 


0.81 


] 

] 

] 

X] 

] 

] 






25 { 


6) 


3.69 


0.71 


J 

) 

] 

XXXXXX] 

] 

] 






24 { 


2) 


3.22 


0.66 


) 

] 

XX] 

] 






23 ( 


6) 


2.80 


0.63 


J 

] 

XXXXXX] 

] 






22 ( 


12) 


2.41 


0.62 


J 

] 

XXXXXXXXXXXX] 

] 






21 ( 


25) 


2.04 


0.60 


] 

) 

XXXXXXXXXXXXXXXXXXXXXXXXX ] 
] 






20 { 


25) 


1.69 


0.59 


] 

XXXXXXXXXXXXXXXXXXXXXXXXX ] 
} 






19 ( 


47) 


1.35 


0.58 


] 

] 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

] 






16 ( 


44) 


1.02 


0.57 


] 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

] 






17 { 


4 9) 


0.70 


0.56 


] 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

) 

] 


124 


0.54(0.1) 0 


16 ( 


37) 


0.40 


0.55 


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ] 

] 


132 


0.31 (0.1)-2 










] 


112 


0.22(0.1) 1 


15( 


41) 


0.10 


0.54 


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

] 

1 


120 


-0.05 (0.1) -2 


14 ( 


34) 


-0.19 


0.53 


J 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

1 


12 


-0.15(0.1) 0 










j 

] 


121 


-0.38(0.1) 1 


13 ( 


21) 


-0.47 


0.53 


XXXXXXXXXXXXXXXXXXXXX ] 
1 


111 


-0.49(0.1) 1 


12 { 


16) 


-0.74 


0.52 


i 

XXXXXXXXXXXXXXXX] 

] 






11 ( 


15) 


-1.01 


0.51 


] 

XXXXXXXXXXXXXXX] 

] 






10 ( 


10) 


-1.27 


0.51 


] 

XXXXXXXXXX] 






9( 


10) 


-1.53 


0.51 


J 

XXXXXXXXXX] 

] 






8 { 


4) 


-1 .80 


0.52 


] 

XXXX] 

] 






7( 


7) 


-2.07 


0.52 


] 

XXXXXXX] 

1 






6< 


5) 


-2.35 


0.54 


J 

XXXXX] 

] 






5( 


0) 


-2.65 


0.56 


J 

] 

] 

] 






4 { 


1) 


-2.99 


0.59 


] 

X] 

] 

1 






3 ( 


1) 


-3.38 


0.66 


J 

] 

X] 

] 

] 






2 ( 


0) 


-3.89 


0.77 


] 

] 

] 
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Appendix C (cont.) 



SCALE: Reading Outcome 

409 PERSONS 4 ITEMS MAX. OF 5 CATEGORIES 16 STEPS 05/06/971 PAGE 10 









MAP SHOWING POSITIONS OF PEOPLE 


AND ITEMS ON THE VARIABLE 






SCORE PERSON 

(FREQ) POSITION 


ERROR 


PEOPLE (N= 409) 


ITEMS (L a 4) 


ITEM 

VALUE (SE) 


FIT 


15 { 0) 


3.70 


1.07 










14 ( 2) 


2.85 


0.82 


X] 








13 ( 4) 


2.27 


0.72 


XX] 








12 ( 18) 


1.78 


0.67 


XXXXXXXXX] 








11 { 20) 


1.34 


0.66 


XXXXXXXXXX] 








10 ( 28) 


0.92 


0.65 


XXXXXXXXXXXXXX] 








9{ 46) 


0.49 


0.66 


XXXXXXXXXXXXXXXXXXXXXXX ] 








8 ( 68) 


0.06 


0.66 


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 


113 123 


0.19 (0.1) 


2-2 










127 131 


-0.16 (0.1) 


0-1 


7 { 42) 


-0.38 


0.67 


XXXXXXXXXXXXXXXXXXXXX ] 








6< 44) 


-0.83 


0.67 


XXXXXXXXXXXXXXXXXXXXXX ] 








5 ( 46) 


-1.29 


0.68 


XXXXXXXXXXXXXXXXXXXXXXX ] 








4 { 36) 


-1.78 


0.70 


XXXXXXXXXXXXXXXXXX ] 








3 ( 22) 


-2.30 


0.75 


XXXXXXXXXXX ] 








2 { 18) 


-2.93 


0.84 


XXXXXXXXX ] 








1 ( 8) 


-3.82 


1.08 


XXXX] 
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Appendix C (cont.) 



SCALE: Writing Outcome 



416 PERSONS 


3 ITEMS 


MAX. OF 5 CATEGORIES 12 STEPS 05/06/97) 


PAGE 


10 






MAP SHOWING POSITIONS OF PEOPLE AND ITEMS ON THE VARIABLE 






SCORE PERSON 

(FREQ) POSITION 


ERROR 


PEOPLE (N= 416) ITEMS (L= 3) 


ITEM 

VALUE (SE) 


FIT 


11 ( 6) 3.69 


1.14 


] 

] 

] 

XXX) 

] 

] 

] 

] 

1 






10 ( 6) 2.67 


0.92 


] 

] 

] 

1 

XXX] 

] 

] 

] 

1 






9 ( 22) 1.90 


0.84 


] 

] 

] 

XXXXXXXXXXX] 

) 

] 

) 






8 ( 32) 1.21 


0.81 


] 

] 

] 

XXXXXXXXXXXXXXXX] 

) 

] 

1 






7 ( 54) 0.57 


0.79 


J 

1 

] 

XXXXXXXXXXXXXXXXXXXXXXXXXXX ] 










) 

] 128 
] 

] 


0.39 (0.1) 


0 


6 ( 78) -0.05 


0.78 


] 

] 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

] 19 122 

] 

] 


-0.24 (0.1) 


0-2 


5 ( 72) -0.66 


0.78 


] 

1 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

) 

] 

1 






4 ( 62) -1.27 


0.78 


J 

J 

] 

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 

] 

] 

1 






3 ( 46) -1.90 


0.81 


J 

] 

] 

XXXXXXXXXXXXXXXXXXXXXXX ] 
] 

] 






2 ( 16) -2.62 


0.89 


] 

] 

] 

XXXXXXXX] 

1 

] 

] 

] 






1 ( 16) -3.59 


1.12 


J 

] 

) 

] 

] 

XXXXXXXX] 
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Appendix C (cont.) 



SCALE: Applied Learning Outcome 

416 PERSONS 3 ITEMS MAX. OF 5 CATEGORIES 12 STEPS 05/06/971 PAGE 10 









MAP SHOWING POSITIONS OF PEOPLE 


AND ITEMS ON THE VARIABLE 




SCORE PERSON 

(FREQ) POSITION 


ERROR 


PEOPLE (N° 416) 


ITEMS (L= 3) 


ITEM 

VALUE (SE) FIT 


11 ( 16) 


3.17 


1.11 


XXXXXXXX) 






10 ( 22) 


2.23 


0.87 


XXXXXXXXXXX] 






9( 34) 


1.56 


0.78 


XXXXXXXXXXXXXXXXX ) 






8( 54) 


0.99 


0.73 


XXXXXXXXXXXXXXXXXXXXXXXXXXX ) 






7 ( 66) 


0.47 


0.71 


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 






6 ( 70) 


CM 

O 

0 

1 


0.70 


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX) 


116 134 


0.09(0.1) 1-1 










13 


-0.24 (0 . 1) -2 


5( 44) 


-0.51 


0.70 


XXXXXXXXXXXXXXXXXXXXXX] 






4 ( 42) 


i 

o 

ISJ 


0.72 


XXXXXXXXXXXXXXXXXXXXX ] 






3 ( 32) 


-1.57 


0.77 


XXXXXXXXXXXXXXXX] 






2( 24) 


-2.22 


0.85 


XXXXXXXXXXXX) 






00 


-3.13 


1.09 


XXXX] 








WHEN THE 


SCORE GROUP 


POSITIONS ARE CLOSER THAN ONE TENTH OF A LOGIT, 


SCORE GROUPS ARE COMBINED: 


X*=NO MORE THAN 2 PERSON/S 
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SCALE: Evaluation Outcome 



422 PERSONS 


5 ITEMS 


MAX. OF 


5 CATEGORIES 20 STEPS 


05/06/ 97 J 


PAGE 11 










MAP SHOWING POSITIONS OF PEOPLE 


AND ITEMS ON THE VARIABLE 




SCORE 


PERSON 










ITEM 


(FREQ) POSITION 


ERROR 




PEOPLE (N= 422) 


ITEMS (L= 5) 


VALUE (SE) FIT 


19< 6) 


3.66 


1.06 




XXX] 






18 ( 10) 


2.83 


0.80 




XXXXX] 






17 < 14) 


2.27 


0. 69 




XXXXXXX] 






16 ( 28) 


1.83 


0.64 




XXXXXXXXXXXXXX] 






15 < 40) 


1.45 


0.60 




XXXXXXXXXXXXXXXXXXXX ] 
} 






14 ( 48) 


1.10 


0.58 




XXXXXXXXXXXXXXXXXXXXXXXX ) 






13 ( 58) 


0.77 


0.56 




XXXXXXXXXXXXXXXXXXXXXXXXXXXXX ] 






12 ( 56) 


0.47 


0.55 




XXXXXXXXXXXXXXXXXXXXXXXXXXXX ] 


126 


0.41 {0 . 1) -2 


11( 48) 


0.18 


0.54 




XXXXXXXXXXXXXXXXXXXXXXXX ] 


15 


0.18 (O.D-5 


10 ( 28) 


i 

o 


0.53 




XXXXXXXXXXXXXX] 


110 


-0.08 (0.1) -4 












14 


-0.24(0.1) 1 












138 


-0.27(0.1) 7 


9< 24) 


-0.38 


0.52 




XXXXXXXXXXXX] 






8 ( 24) 


-0.65 


0.52 




XXXXXXXXXXXX] 






7{ 12) 


-0.92 


0.52 




XXXXXX] 






6{ 8) 


-1.19 


0.53 




XXXX] 






5< 4) 


-1.48 


0.55 




XX] 






4 { 4) 


-1.80 


0.58 




XX] 






3 ( 0) 


-2.17 


0.64 










2 ( 0) 


-2.65 


0.75 










1{ 0) 


-3.40 


1.03 
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